The present invention relates generally to processing signals, and more particularly to a system for the rapid compression of hypersensor data sets that contain objects, substances, or patterns embedded in complex backgrounds. A hypersensor is a sensor which produces as its output a high dimensional vector or matrix consisting of many separate elements, each of which is a measurement of a different attribute of the system or scene under construction. A hyperspectral imager is an example of a hypersensor. Hypersensors based on acoustic or other types of signals, or combinations of different types of input signals are also possible.
Historically there have been three types of approaches to the problems relating to the detection of small, weak or hidden objects, substances or patterns embedded in complex backgrounds.
The first approach has been to use low dimensional sensor systems which attempt to detect a clean signature of a well known target in some small, carefully chosen subset of all possible attributes, e.g., one or a few spectral bands. These systems generally have difficulty when the target signature is heavily mixed in with other signals, so they typically can detect subpixel targets or minority chemical constituents of a mixture only under ideal conditions, if at all. The target generally must fill at least one pixel, or be dominant in some other sense as in some hyperspectral bands. Also, the optimal choice of bands may vary with the observing conditions or background (e.g. weather and lighting), so such systems work best in stable, predictable environments. These systems are simpler than the high dimensional sensors (hypersensors), but they also tend to be less sensitive to subdominant targets and less adaptable.
The second approach has been to employ high dimensional sensor systems which seek to detect well known (prespecified) targets in complex backgrounds by using Principle Components Analysis (PCA) or similar linear methods to construct a representation of the background. Orthogonal projection methods are then used to separate the target from the background. This approach has several disadvantages. The methods used to characterize the background are typically not `real time algorithms`; they are relatively slow, and must operate on the entire data set at once, and hence are better suited to post-processing than real time operation. The background characterization can get confused if the target is present in a statistically significant measure when the background is being studied, causing the process to fail. Also, the appearance of the target signature may vary with the environmental conditions: this must be accounted for in advance, and it is generally very difficult to do. Finally, these PCA methods are not well suited for detecting and describing unanticipated targets, (objects or substances which have not been prespecified in detail, but which may be important) because the representation of the background constructed by these methods mix the properties of the actual scene constituents in an unphysical and unpredictable way. PCA methods are also used for compression schemes however they have many of the same shortcomings. Learned Vector Quantization (LVQ) is also used for compression. Current LVQ schemes use minimum noise fraction (MNF) or average patterns of PCs to compress the data, or various iterative methods which are slow and require a priori knowledge of sensor characteristics.
The more recent approach is based on conventional convex set methods, which attempt to address the `endmember` problem. The endmembers are a set of basis signatures from which every observed spectra in the dataset can be composed in the form of a convex combination, i.e., a weighted sum with non-negative coefficients. The non-negativity condition insures that the sum can sensibly be interpreted as a mixture of spectra, which cannot contain negative fractions of any ingredient. Thus every data vector is, to within some error tolerance, a mixture of endmembers. If the endmembers are properly constructed, they represent approximations to the signature patterns of the actual constituents of the scene being observed. Orthogonal projection techniques are used to demix each data vector into its constituent endmembers. These techniques are conceptually the most powerful of the previous approaches, but prior methods for implementing the convex set ideas are slow, (not real time methods) and cannot handle high dimensional pattern spaces. This last problem is a serious limitation, and renders these methods unsuitable for detecting weak targets, since every constituent of a scene which is more dominant than the target must be accounted for in the endmember set, making weak target problems high dimensional. In addition, current convex set methods give priority to the constituents of the scene which are dominant in terms of frequency of occurrence, with a tendency to ignore signature patterns which are clearly above the noise but infrequent in the data set. This makes them unsuitable for detecting strong but small targets unless the target patterns are fully prespecified in advance.
When operating in high dimensional pattern spaces massive quantities of data must be managed which requires hundreds of millions of computations for each pixel. Thus the need to compress massive quantities of data for storage, download, and/or real time analysis becomes increasingly important and equally elusive.